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CHAPTER  I 


INTRODUCTION 

Overview 

One  of  the  most  significant  items  of  continuing  con¬ 
cern  to  the  Department  of  the  Air  Force  is  the  need  for 
improved  reliability  and  maintainability  of  its  weapon  sys¬ 
tem's  equipment  (17:1).  In  an  effort  to  achieve  this  end, 
the  Air  Force,  in  FY  1969,  first  applied  the  concept  of  the 
Reliability  Improvement  Warranty  (RIW)  to  the  acquisition 
process  (19-.N-3).  In  1974,  in  response  to  a  request  from 
the  Office  of  the  Secretary  of  Defense,  the  Air  Force  began 
a  formal  trial  program  of  the  RIW  concept. 

RIW  is  a  contractual  technique  used  in  the  commer¬ 
cial  environment  and  currently  being  utilized  on  a  trial 
basis  within  the  Department  of  Defense  (DOD)  as  a  means  of 
implementing  such  improvements  as  greater  equipment  reli¬ 
ability,  reduced  repair  costs,  and  lower  life  cycle  costs. 
This  was  previously  known  as  a  "Failure  Free"  or  "Standard" 
Warranty.  The  objective  of  a  RIW  is  to  motivate  and 
encourage  contractors  to  design  and  produce  equipment  which 
will  have  a  low  failure  rate  as  well  as  low  repair  costs 
after  failure  due  to  field/operational  use.  Furthermore, 
this  technique  attempts,  through  the  use  of  contractual 
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agreements  (where  the  period  of  performance  extends  over 
several  years) ,  to  provide  an  incentive  for  contractors  to 
improve  the  reliability  of  their  equipment  and  to  reduce 
repair  costs  during  the  period  of  warranty  coverage  in 
order  to  maximize  their  profits  (17:1-2). 

Under  the  concept,  the  contractor  is  obligated  to 
repair  all  warranted  items,  excluding  exceptions  stated  in 
the  contract,  for  the  duration  of  the  warranty  period.  In 
return  for  providing  repairs,  the  contractor  receives  a 
firm-fixed  fee.  The  fee,  which  is  determined  prior  to  award 
of  the  contract,  does  not  vary  with  the  number  of  units  sent 
to  the  contractor  for  repair.  If  few  repairs  are  required 
over  the  life  of  the  warranty,  the  contractor's  profits 
would  likely  be  high.  Conversely,  a  high  item  failure  rate 
would  involve  a  large  number  of  repairs  causing  the  con¬ 
tractor's  repair  costs  to  exceed  the  fixed  fee.  The  fee, 
therefore,  is  designed  to  provide  the  contractor  an  incen¬ 
tive  to  manufacture  more  reliable  items  requiring  fewer 
repairs.  The  contractor  is  authorized  some  latitude  to 
change  the  item  during  the  warranty  period  to  improve  the 
item's  reliability  (17:2).  Government  contracting  officers 
anticipate  that  continued  emphasis  by  civilian  contractors 
to  decrease  failure  rates  will  bring  about  benefits  for  the 
government  in  the  form  of  better  equipment  availability  and 
lower  life  cycle  cost  (17:29). 


The  RIW  contract  clause  requires  the  contractor  to 
maintain  the  items  under  warranty  for  the  duration  of  the 
warranty  period.  The  length  of  the  warranty  period  is 
formally  stated  in  the  RIW  clause  (17:2).  RIW  warranty 
periods  have  varied  from  a  minimum  of  two  years  to  a  maxi¬ 
mum  of  six  years  and  have  averaged  over  three  years  in 
length  (17:48) . 

It  is  important  to  note  that  RIW  is  not  a  mainte¬ 
nance  contract  and  does  not  require  that  the  contractor 
provide  routine  periodic  upkeep,  regulation,  adjusting, 
cleaning,  or  other  normal  maintenance.  A  RIW  also  does  not 
cover  components  of  a  warranted  item  which  are  expected  to 
need  replacement  under  normal  use  during  the  term  of  the 
warranty  (such  as  filters,  light  bulbs,  etc.).  These  items 
may  be  provided  for  by  separate  provisions  in  the  contract 
consistent  with  current  laws  and  regulations,  but  they  shall 
not  be  included  in  the  RIW  provision.  Furthermore,  if  the 
item  failure  can  be  attributed  to  obvious  abuse,  aircraft 
crash,  tampering,  or  other  reason  beyond  the  control  of  the 
contractor,  repair  of  the  item  is  not  without  cost  to  DOD 
(17:16).  In  general,  a  RIW  will  provide  for  the  repair  or 
replacement  of  failed  units  as  well  as  agreed  to  "no-cost" 
engineering  changes  and  the  calibration,  adjustment  and 
associated  testing  therewith  (17:2). 


Background 

Life  cycle  cost  is  the  basis  of  the  RIW  concept.  The 
object  of  life  cycle  cost  is  to  insure  the  lowest  overall 
ownership  cost  to  the  government  during  the  life  of  the  hard¬ 
ware  (11:1-1).  The  declining  purchasing  power  of  the  defense 
budget  dictates  a  need  to  explore  ways  to  cut  the  ownership 
costs  associated  with  weapon  systems.  In  an  attempt  to 
answer  this  need,  RIW  is  currently  being  tested  within  DOD 
for  its  potential  to  minimize  these  costs  (18:30). 

The  Army  and  Navy  studied  the  commercial  airlines 
industry's  contracting  techniques  because  of  their  past 
ability  to  acquire  more  reliable  equipment  than  the  military 
services.  During  this  study,  it  was  noted  that  the  airlines 
employed  the  extensive  use  of  warranties.  As  a  result,  the 
Navy  began  to  apply  this  lesson  learned  from  the  airlines  by 
developing  a  warranty  concept  which  later  became  known  as 
RIW  (3 : B-2  to  B-5) . 

The  Office  of  the  Secretary  of  Defense,  in  August 
1974,  requested  tri-service  experimentation  with  RIWs.  Each 
service  was  directed  to  establish  procedures  for  identifying 
potential  contracts  in  which  RIW  could  be  incorporated. 
Following  application  of  RIW  to  selected  programs/contracts , 
the  services  were  to  evaluate  the  RIW  concept  as  a  potential 
contracting  technique  (8:v). 
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In  response  to  the  request  of  the  Office  of  the 
Secretary  of  Defense,  the  Air  Force  began  applying  the  con¬ 
cept  of  RIW  to  selected  elements  of  programs  which  were  in 
the  early  stages  of  development.  With  one  exception,  appli¬ 
cation  within  the  Air  Force  has  been  limited  to  avionics 
equipment.  There  are  currently  five  programs  in  the  Air 
Force  that  involve  a  major  application  of  RIW.  A  major 
application  of  the  concept  is  one  in  which  RIW  has  been 
included  in  the  contract  since  the  early  stages  of  develop¬ 
ment  and  has  not  been  subsequently  deleted  from  the  con¬ 
tract  (14) .  These  programs  are  listed  in  Appendix  A.  Other 
Air  Force  programs  which  employed  the  RIW  concept  in  some 
stage  of  program  development,  but  are  not  major  applications 
are  listed  in  Appendix  B. 

In  July  1978,  Hq  USAF/RDC  established  Hq  AFLC/LOM  as 
the  Air  Force  Warranty  Information  Center  (WIC)  data  reposi¬ 
tory  for  RIW.  The  following  generic  data  were  to  be 
retained  in  the  WIC  (11) : 

1.  Analysis  documentation  to  support  determination 
to  apply  RIW  on  specific  programs. 

2.  Periodic  contractor  RIW  reports. 

3.  RIW  administration  as  prescribed  by  AFLC  Test 
Regulation  800-7. 

4.  Follow-on  support  determinations  to  ascertain 
the  logistic  support  concept  after  completion  of  RIW. 
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The  contractors'  reports  are  not  standardized  in 
format,  content,  or  frequency.  These  factors  are  determined 
by  the  individual  program  contracts.  The  report  frequencies 
range  from  monthly  to  semiannually  and  include  a  summary, 
for  the  reporting  period,  of  the  number  of  units  returned 
to  the  factory,  the  status  of  the  units  upon  contractor 
receipt,  contractor's  action,  reshipraent  date,  compilation 
of  mean  time  between  failures,  and  pipeline  performance 
(14)  . 

The  purpose  of  the  WIC  is  to  collect  and  store  data 
on  RIW  programs.  No  analysis  of  the. data  for  content  or 
completeness  is  done  at  the  WIC.  Therefore,  there  is  no 
record  at  the  WIC  to  indicate  whether  or  not  the  data  con¬ 
tained  in  the  repository  fulfills  the  requirements  of 
Hq  USAF/RDC  (14) . 

According  to  an  Air  Force  Audit  Agency  report,  a 
master  plan  to  evaluate  the  impact  of  the  RIW  concept  on 
Air  Force  programs  and  bring  the  trial  program  to  a  con¬ 
clusion  had  not  been  prepared.  An  office  of  primary  respon¬ 
sibility  (OPR)  to  prepare  a  master  plan  and  perform  the 
evaluation  has  not  been  established  (2:10-11). 
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CHAPTER  II 


PROBLEM  DEFINITION 

Evaluation  Progress 

A  review  of  the  literature  revealed  the  following 
significant  facts  concerning  the  evaluation  of  RIW: 

1.  Trial  use  began  in  1974  (8:v). 

2.  Five  programs  involving  major  applications  of 
RIW  have  been  established  (14) . 

3.  A  WIC  was  established  in  1978  to  collect  RIW 

data. 

4 .  An  evaluation  plan  for  the  RIW  concept  has  not 
been  established. 

5.  An  OPR  to  develop  an  evaluation  plan  has  not 
been  identified. 

Therefore,  it  appears  that  some  data  is  being  col¬ 
lected  but  without  a  plan  for  its  use  or  evaluation.  Fur¬ 
thermore,  it  is  not  known  whether  this  data  is  complete 
(i.e.,  that  all  the  data  requested  has  been  received  by  the 
repository) . 

Evaluation  Need 

As  shown  in  Appendix  A,  the  table  of  programs 
involving  major  applications  of  RIW,  the  warranty  period  for 
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the  first  program  will  expire  in  October  1980.  The  last 
program's  warranty  expires  in  January  1983.  Since  a  poten¬ 
tial  item  to  evaluate  might  be  equipment  performance  follow¬ 
ing  the  warranty  period,  it  is  anticipated  that  it  will  be 
possible  to  draw  conclusions  from  the  trial  in  FY  1984  (14) . 
However,  in  order  to  do  this,  an  evaluation  plan  needs  to 
be  developed  to  identify  which  factors  are  important  in  the 
evaluation  of  RIW.  The  need  exists  to  identify  any  factors 
common  to  the  five  programs  using  major  applications  of  RIW 
and  determine  whether  any  generalizations  can  be  made  from 
the  experience  of  these  five  programs  for  the  use  of  RIW 
in  future  programs. 


Justification 

The  progress  of  the  evaluation  program  points  out 
the  need  for  measurable  parameters  to  evaluate  the  RIW  con¬ 
cept.  This  need  is  clearly  recognized  within  the  Air  Force. 
A  17  January  1978  letter  from  USAF/LG  states. 

Inputs  from  all  Major  Commands  involved  in  RIW 
are  needed.  The  evaluation  should  address  "cause 
and  effect"  relationships  and  draw  significant  con¬ 
clusions  from  the  Air  Force  test.  Parameters  must 
be  developed  to  bring  the  RIW  trial  from  an  open- 
ended  test  to  a  meaningful  conclusion.  The  evaluation 
results  should  be  the  foundation  for  establishing  a 
corporate  Air  Force  position  on  the  merits  of  the 
RIW  concept  [10]. 


Problem  Statement 

RIW  is  currently  under  a  trial  program  which  is 
subject  to  an  evaluation  in  FY  1982.  A  comprehensive  plan 
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to  evaluate  the  impact  of  the  RIW  concept  on  Air  Force 
programs  and  bring  the  trial  program  to  a  conclusion  has 
not  been  prepared.  This  highlights  an  immediate  need  for 
a  list  of  criteria  to  measure  effectiveness  of  the  RIW  con¬ 
cept.  Identification  of  such  criteria  would  provide  a 
foundation  for  development  of  a  master  plan  to  evaluate 
the  RIW  concept  in  the  Air  Force. 

Research  Objective 

The  objective  of  this  study  is  to  formulate  a  set 
of  criteria  from  among  the  current  RIW  programs  that  can  be 
incorporated  into  an  evaluation  plan  for  the  RIW  concept. 

Research  Questions 

1.  What  elements  are  affected  by  RIW  in  Air  Force 
programs  that  use  it? 

2.  Which  of  these  elements  are  most  important  to 
evaluating  the  effect  of  RIW? 

Scope 

To  limit  this  research  effort,  only  the  five  pro¬ 
grams  involving  major  applications  of  RIW  will  be  addressed 
in  the  data  collection  and  analysis.  The  use  of  the  RIW 
concept  in  the  other  programs  was  either  not  employed  at 
the  beginning  of  the  contracting  process,  or  RIW  was  applied 
to  a  contract  and  subsequently  determined  to  be 
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inappropriate  to  the  program  (14) .  Therefore,  the  informa¬ 
tion  on  these  programs  is  limited. 


Furthermore,  it  is  not  the  intent  of  this  research 
effort  to  establish  a  complete  evaluation  plan  for  the  RIW 
concept  because  that  would  take  far  more  time  and  resources 
than  this  thesis  term  has  available.  This  effort,  then, 
will  be  limited  to  the  identification  of  elements  of  the 
RIW  concept  upon  which  development  of  an  evaluation  and 
data  collection  plan  can  be  based. 

The  intent  of  this  research  effort  is  to  establish 
what  progress  has  been  made  in  the  Air  Force  evaluation  and 
what  direction  should  be  taken  in  the  future.  This  chapter 
discusses  the  concept  of  RIW  and  its  objectives  followed  by 
a  development  of  the  progress  of  the  Air  Force  program 
established  to  evaluate  the  RIW  concept. 

Chapter  II  establishes  the  need  and  justification 
for  an  evaluation  program.  The  problem  is  identified  along 
with  the  limitations  of  this  research  effort. 

Chapter  III  establishes  the  plan  to  collect  and 
analyze  the  criteria  of  the  RIW  concept  around  which  an 
evaluation  plan  can  be  developed. 

Chapter  IV  contains  analysis  of  the  data  collected. 
The  thesis  closes  with  a  discussion  of  conclusions  and 
recommendations  in  Chapter  V. 


CHAPTER  III 


RESEARCH  METHOD 

Overview 

An  important  aspect  of  any  research  effort  is  the 
method  employed  by  the  researcher.  This  chapter  describes 
in  detail  the  specific  procedure  used  in  the  research  design 
including  data  acquisition  and  data  analysis. 

Preliminary  research  indicated  that  no  common  pool 
of  knowledge  about  the  separate  applications  of  RIW  exists 
(10) .  There  was  a  need  to  identify,  within  the  five  pro¬ 
grams,  elements  which  were  affected  by  RIW.  To  accomplish 
this,  a  method  was  needed  to  bring  the  experience  of  RIW 
within  the  five  programs  together  to  form  a  common  pool  of 
knowledge.  A  consensus  of  opinion  was  one  method  available 
to  form  this  common  pool  of  knowledge.  A  useful  tool  to 
develop  successful  group  decisions  when  no  one  individual 
possesses  all  of  the  knowledge  available  is  the  Delphi 
Technique.  This  method  of  evaluating  and  tabulating  inde¬ 
pendent  opinions  without  group  discussions  was  appropriate 
to  develop  the  consensus  needed  for  this  research  effort. 

Delphi  Method 

A  Business  Horizons  article  describing  the  Delphi 
Technique,  states,  "Group  decisions  are  necessary  when  the 
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scope  of  a  problem  is  such  that  no  individual  has  sufficient 
expertise  and  knowledge  to  effect  a  solution  [1:51]." 

It  can  be  seen  from  Appendix  D  that,  within  each 
program,  two  different  individuals  are  knowledgeable  about 
RIW.  Each  of  these  individuals  knows  the  concept  as  it 
applies  to  his  particular  program.  Although  an  OPR,  the 
WIC,  has  been  established  for  data  collection,  there  is  no 
OPR  in  which  one  individual  is  aware  of  the  impact  of  RIW 
on  all  five  programs  (14)  .  For  this  reason,  it  appeared 
that  a  group  decision  was  necessary.  The  Delphi  Technique 
was  first  employed  approximately  twenty-five  years  ago  with 
an  Air  Force  sponsored  program.  The  method  was  used  to: 

.  .  .  collect  intuitive  judgments  of  a  group  of 
experts  in  the  subject  area  of  interest  with  many 
unknowns .  This  technique  was  probably  superior  to 
rational  extrapolations  of  known  trend  [4:20]. 

This  research  employed  an  identical  procedure  as  was  used 

with  the  first  Delphi  application  except  the  subject  matter 

of  interest  was  RIW  (see  Appendix  C) . 

Advantages  and  Characteristics 

The  Delphi  method  offered  many  advantages  for  this 
research  effort. 

1.  It  allowed  experts  from  diverse  backgrounds  to 
make  decisions.  In  this  situation,  the  diverse  backgrounds 
of  the  experts  were  considered  to  be  their  experience  in  the 
five  different  programs. 
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2.  The  interaction  among  experts  was  controlled  by 
the  researchers.  Telephone  interviews  were  used  for  data 
collection.  The  telephone  interview  followed  by  controlled 
feedback  of  interview  results  offered  an  excellent  means  of 
controlling  interaction  among  respondents. 

3.  The  experts  were  not  identified  to  each  other  in 
any  way.  Geographical  separation,  coupled  with  interviewer 
control,  offered  the  necessary  anonymity. 

4.  The  anonymity  eliminated  bias  from  each  expert. 

5.  The  halo  effect  and  bandwagon  effect  were 
virtually  eliminated.  This  was  a  direct  result  of  the 
anonymity  discussed  in  3. 

6.  A  consensus  of  opinion  was  formed  by  requiring 
justification  for  any  significant  deviation  from  the  group 
average.  Consensus  was  achieved  through  continuous  screen¬ 
ing  and  feedback  of  information  from  previous  interviews 
(1:51-52)  . 

Use  of  the  telephone  interview  minimized  problems  which 
could  have  been  induced  by  the  geographical  dispersion  of 
panel  experts.  Use  of  the  telephone  interview  also  reduced 
the  demands  placed  upon  the  respondents. 

Procedure 

The  steps  involved  in  the  Delphi  method  can  vary 
somewhat  based  on  the  application.  The  steps  used  here 
included  the  following: 


1.  Selection  of  panel  experts. 


2.  Initial  contact  of  panel  experts. 

3.  Initial  questionnaire/interview. 

4.  Follow-up  questionnaires/interviews. 

Selection  of  Panel  Experts.  The  panel  consisted  of 
individuals  whose  judgments  were  being  sought  and  who  also 
agreed  to  participate  in  the  project.  The  composition  of 
a  panel  of  experts  is  critical  to  the  Delphi  Technique 
effectiveness.  Accordingly,  four  criteria  were  used  in 
choosing  the  experts. 

1.  They  were  required  to  have  a  basic  knowledge  of 
the  problem  area  and  to  be  able  to  apply  that  knowledge. 

2.  A  good  performance  record  in  the  participant's 
particular  area  of  expertise  was  required. 

3.  They  had  to  have  the  time  available  to  partici¬ 
pate  to  the  conclusion  of  the  program. 

4.  All  participants  had  to  be  willing  to  give  the 
amount  of  time  and  effort  to  do  a  thorough  job  of  partici¬ 
pation  (5 : 271) . 

During  initial  contact  with  the  individuals  it  was 
determined  how  long  the  individuals  had  been  working  in 
their  present  capacity.  It  was  assumed  that  two  years  in 
the  position  of  program  manager  or  contracting  officer  on 
any  of  the  five  programs  would  yield  the  experience  neces¬ 
sary  to  consider  the  individuals  knowledgeable  of  RIW  and, 
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thereby,  qualified  under  criterion  1,  above-  If  the  indi¬ 
viduals  had  worked  on  the  program  two  years  in  their  present 
position,  it  was  assumed  that  longevity  and  position  quali¬ 
fied  the  respondents  under  criterion  2.  Criteria  3  and  4 
qualifications  were  established  during  the  initial  contact. 
Due  to  the  knowledge  and  composition  of  the  panel  experts, 
there  was  homogeneity  among  the  experts.  Therefore  the 
number  of  experts  was  limited  to  ten  (5:273). 

Contacting  Panel  Experts.  The  next  step  in  the 
process  was  to  contact  the  people  who  had  been  nominated 
during  the  selection  process.  The  information  that  was 
conveyed  to  the  experts  included  the  purpose  of  the  study, 
the  primary  question/problem  to  be  analyzed,  their  role  in 
reaching  a  solution,  the  importance  of  a  concerted  effort 
on  their  part  to  insure  successful  results,  and  the  unique¬ 
ness  of  their  abilities  in  the  total  effort  Cl: 54). 

The  First  Questionnaire/Interview.  Appendix  F  con¬ 
tains  the  guide  used  to  conduct  the  first  interview.  The 
primary  consideration  in  structuring  the  first  interview  was 
to  eliminate  potential  researcher  bias  obtained  through  the 
literature  search.  Balancing  this  consideration  was  the 
concern  for  allowing  enough  structure  to  the  interview  to 
stimulate  the  interviewee's  thought  processes  on  the  subject. 
In  other  words,  an  interviewee's  response  to  one  very 
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general  open-ended  question  might  be  unclear  and  incomplete 
while  an  interview  structured  around  a  list  of  specific 
criteria  might  result  in  a  more  meaningful  response.  There¬ 
fore,  the  interview  was  structured  around  a  set  of  general 
elements  of  RIW  which  might  lead  to  identifying  specific 
criteria.  Since  this  list  of  criteria  might  not  be  all- 
inclusive,  an  open-ended  question  was  developed  to  gather 
additional  elements  which  the  interviewees  believed  to  be 
important  to  their  programs.  The  basic  assumption  was  that 
a  discussion  centered  around  the  list  of  general  elements 
would  stimulate  more  complete  discussion  and  responses  to 
the  open-ended  question. 


Follow-up  Questionnaires/Interviews .  The  second 
questionnaire  was  a  compiled  list  of  perceived  criteria 
specified  by  the  panelists.  The  respondents  were  provided 
the  opportunity  to  apply  a  rank  order  scale  to  this  list  of 
criteria.  A  rank  order  scale  was  used  because: 

.  .  .  rank  order  scales  have  two  convenient  analytic 
advantages.  One,  the  scales  of  individuals  can  easily 
be  intercorrelated .  Two,  they  partially  escape  response 
set  and  the  tendency  to  agree  with  socially  desirable 
items  [1  3:  505]  . 

Using  this  scale  each  respondent  was  asked  to  rank  each 
criteria  according  to  his/her  perception  of  its  importance. 
For  example,  from  the  nominated  list  of  criteria,  the 
respondent  was  asked  to  rank  the  criteria  from  most 
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important  (1)  to  least  important  (11)  .  Since  respondents 
answer  each  item,  the  rank  order  scale  is  more  reliable 
than  other  scales  which  do  not  require  responses  for  each 
item.  It  is  also  easy  to  use  in  stimulus-centered  studies. 
If  time  and  cost  limitations  require,  rank  order  scales  can 
be  developed  in  an  arbitrary  manner  (6:248-250). 

Each  expert  ranked  each  nominated  criterion  on  a 
scale  from  one  to  eleven.  This  scale  was  found  to  be  quite 
common  and  used  in  similar  situations  (16:418-420).  A  com¬ 
posite  of  responses  to  the  second  questionnaire  along  with 
an  explanation  of  any  misconceptions  on  the  part  of  any 
expert  formed  the  basis  for  follow-on  interviews. 

The  Delphi  process  requires  a  measure  of  central 
tendency.  Central  tendency  was  determined  by  the  amount  of 
agreement  between  rank  ordered  lists  of  criteria.  The  sta¬ 
tistic  used  to  measure  agreement  on  this  data  was  the 
Kendall's  Tau.  The  Tau  statistic  is  an  indicator  of  the 
agreement  that  exists  between  two  rank  ordered  lists 
(9:284-294).  The  Tau  statistic  was  computed  for  each  pair¬ 
wise  combination  of  rank -ordered  lists.  Each  Tau  value  was 
then  tested  at  the  a  =  .05  level  of  significance  to  deter¬ 
mine  which  participants  had  reached  a  consensus  regarding 
the  rank  ordering  of  the  criteria. 

Each  successive  round  of  the  Delphi  process  con¬ 
sisted  of  tabulated  results  of  the  previous  round.  These 


rounds  allowed  each  respondent  to  compare  his  rankings  with 
those  of  the  other  participants.  Each  participant  was 
asked  to  reevaluate  his  rankings  based  upon  the  rankings  of 
other  participants. 

The  primary  purpose  of  this  information  feedback  is 
to  produce  more  precise  results  and  encourage  opinion  con¬ 
vergence  (1:55).  Although  the  number  of  questionnaires 
varies,  it  is  usually  conceded  that  at  least  three  but  less 
than  seven  are  necessary  for  a  reasonable  consensus  of 
opinion  (5:272).  See  Appendix  C  for  a  schematic  of  the 
Delphi  Technique. 


Limitations 

The  crucial  step  in  the  entire  process  involves  the 
selection  of  the  panel  of  experts.  The  mechanics  of  the 
Delphi  can  be  completely  negated  by  poor  panel  selection 
and  poor  motivation  (1:55).  The  motivation  aspect  can  go 
through  a  filtering  device  (contacting  the  panel  experts) 
which  occurs  prior  to  the  first  round.  Any  expert  who  does 
not  seem  interested  in  the  project  was  eliminated  as  a 
potential  panel  member.  This  process  of  accepting  or 
rejecting  an  expert  was  the  sole  decision  of  the  researchers. 

The  aspect  of  poor  panel  selection  is  a  more  delicate 
area.  It  was  assumed  the  participants  involved  with  the 
current  RIW  program  were  experts  because  of  their  length  of 
time  involved  with  the  program  (at  least  two  years) .  The 


18 


experts  were  also  assumed  to  be  qualified  based  on  their 
position  which  allowed  them  access  to  vast  amounts  of  infor¬ 
mation  on  the  RIW  program. 

The  length  of  time  required  for  the  analysis  also 
presented  a  potential  problem.  This  problem  was  alleviated 
by  gathering  the  results  of  the  questionnaires  over  the 
telephone  in  lieu  of  the  results  mailed  to  the  researchers. 

Source  of  Data 

Population — The  population  consisted  of  all  persons 
associated  with  RIW  programs  past  and  present. 

Sample — The  sample  was  drawn  from  existing  RIW  pro¬ 
grams.  The  experts  were  drawn  from  contracting  officers 
and  program  managers  listed  in  Appendix  D. 

Operational  Definition — The  results  of  the  final 
round  of  Delphi  yielded  the  operational  definition  of  the 
perceived  opinions.  The  opinions  were  not  considered  opera¬ 
tionalized  until  consensus  was  formulated. 

Data  Collection  Plan  and  Analysis 

As  was  previously  mentioned,  the  method  of  gathering 
data  was  a  combination  of  questionnaire  and  interview.  The 
exact  content  of  questions  (proposed  criteria)  were  deter¬ 
mined  by  the  responses  of  panel  experts  to  the  previous 
questionnaire (s) .  This  established  list  of  possible  cri¬ 
teria  (results  from  round  one)  were  fed  back  to  the 
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participants  who  were  asked  to  rank  each  nominated  cri¬ 
terion  on  a  rank  order  scale.  Upon  receipt,  the  ratings 
were  tabulated  and  analyzed  using  a  Statistical  Package  for 
the  Social  Sciences  (SPSS)  subprogram,  NONPAR  CORR  to  com¬ 
pute  Kendall's  Tau  statistic.  For  the  purpose  of  this 
research  effort,  a  level  of  significance  of  a  =  0.05  was 
used  to  interpret  the  results  of  this  test.  Based  upon  the 
analysis,  a  determination  was  made  as  to  which,  if  any, 
pairs  of  respondents  had  reached  a  consensus.  The  subse¬ 
quent  rounds  of  questionnaires  contained  the  tabulated 
ratings  from  the  entire  sample.  The  criteria  which  pre¬ 
cluded  a  consensus  required  justification  from  the  respond¬ 
ents.  For  example,  it  was  anticipated  that  if  one 
respondent  ranked  a  criterion  as  extremely  important,  and 
another  ranked  the  same  criterion  as  extremely  unimportant, 
each  was  to  provide  an  explanation  of  any  misconception. 
These  explanations,  coupled  with  the  tabulated  results  of 
the  previous  round,  were  summarized  and  distributed  in  the 

succeeding  rounds  so  that  other  participants  could  take 

« 

exception  to  a  stated  position.  The  purpose  of  information 
feedback  was  to  produce  more  precise  responses  and  encourage 
opinion  convergence  (1:55). 


20 


L 


CHAPTER  IV 


DATA  COLLECTION  AND  ANALYSIS 

This  chapter  summarizes  and  evaluates  the  data  col¬ 
lected  during  each  round  of  interviews.  For  each  round  of 
interviews,  a  short  summary  and  evaluation  is  presented  to 
relate  the  finding  to  the  next  round  of  interviews. 

Round  I 

During  initial  data  collection  all  panel  experts 
were  highly  interested  in  their  respective  programs  and 
expressed  an  earnest  desire  to  participate. 

The  collection  of  data  during  round  I  was  obtained 
by  responses  to  the  premailed  interview  questionnaire  con¬ 
tained  in  Appendix  E.  As  noted  in  Chapter  III,  there  exists 
no  common  pool  of  knowledge  about  the  five  existing  RIW  pro¬ 
grams.  Therefore,  the  researchers,  through  their  literature 
search,  nominated  eleven  criteria  that  appeared  to  be  common 
to  current  programs.  The  following  is  a  list  of  the  eleven 
criteria: 

contractor  competition 
acquisition  cost 
equipment  limitation 
initial  support  cost 


logistic  support  cost 

reliability 

maintainability 

number  of  spares  required 

maintenance  skill  required 

support  manpower 

equipment  modification 

During  telephone  interviews,  each  participant  was  asked  to 
respond  to  the  set  of  nominated  criteria  using  the  struc¬ 
tured  interview  guide  contained  in  Appendix  F . 

In  Appendix  G,  individual  responses  to  the  questions  • 
in  interview  I  are  coded  into  a  format  of  Y  Cyes) ,  N  (no) , 
and  U  (undecided  or  don't  know).  Results  of  this  round  were 
used  to  narrow  the  criteria  list  to  those  evaluation  cri¬ 
teria  that  were  common  to  existing  programs.  The  Delphi 
process  dictates  that  if  all  (or  a  predetermined  number  of) 
participants  fail  to  respond  (or  respond  negatively)  to  any 
criterion,  it  should  be  eliminated  from  future  consideration 
(1:54)  . 

As  shown  in  Appendix  G,  fewer  than  three  inter¬ 
viewees  responded  with  "yes"  to  four  of  the  criteria  ini¬ 
tially  nominated — logistical  support  cost,  equipment  limi¬ 
tation,  initial  support  costs,  and  support  manpower.  Using 
the  consensus  principle  of  the  Delphi  method,  these  four 
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criteria  were  determined  to  be  inappropriate  for  considera¬ 
tion  in  round  II,  and  consequently,  deleted  from  the  list. 

Results  of  this  round  were  also  used  to  supplement 
the  list  with  additional  criteria  that  were  not  included  in 
the  initial  list.  The  accomplishment  of  this  portion  of 
the  Delphi  process  was  achieved  through  the  use  of  an  open- 
ended  question  (see  Appendix  F) .  The  participants  nomi¬ 
nated  the  following  list  of  criteria  which  they  perceived 
to  be  important  in  accurately  assessing  the  impact  of  RIW 
on  existing  programs:  data  collection,  effect  of  RIW  on  the 
supply  system,  RIW  contract  clause,  and  equipment  use  rate. 

Round  I  Analysis 

The  Delphi  process  is  based  upon  the  principle  of 
using  feedback  from  previous  rounds  to  move  toward  a  con¬ 
sensus  in  successive  rounds  (1:55).  Therefore,  the  two 
types  of  anonymous  feedback  from  round  I  (consensus  on 
inapplicability  of  four  criteria  on  the  initial  list  and 
suggestion  of  four  additional  criteria)  provided  the  founda¬ 
tion  for  constructing  the  list  of  criteria  to  be  evaluated 
in  round  II: 

acquisition  cost 
contractor  competition 
data  collection 

effect  of  RIW  on  the  supply  system 
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equipment  modification 

maintainability 

maintenance  skill  required 

number  of  spares  required 

reliability 

RIW  contract  clause 

use  rate 


Round  II 

The  second  round  of  interviews  provided  the  partici¬ 
pants  their  first  opportunity  to  rank  the  RIW  rating  cri¬ 
teria.  The  eleven  criteria  ranked  during  this  round  were 
collected  and  compiled  from  the  results  of  round  I.  Prior 
to  contact  via  telephone,  each  interviewee  received  a  copy 
of  the  interview  II  schedule  as  depicted  in  Appendix  H. 

Each  participant  was  asked  to  consider  the  signifi¬ 
cance  that  each  criterion  would  have  in  assessing  the  over¬ 
all  importance  of  RIW  to  his  program.  Using  this  perceived 
level  of  significance,  the  interviewee  was  asked  to  rate 
each  criterion  with  "1"  representing  the  most  important  and 
"11"  representing  the  least  important  criterion. 

During  the  telephone  interview,  each  participant's 
responses  were  recorded.  Upon  completion  of  all  telephone 
interviews  for  round  II,  the  responses  of  all  participants 
were  tabulated  as  depicted  in  attachment  I.  This  tabulation 
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provides  a  two  variable  matrix  format  of  the  interview 
results;  the  interviewee  (number)  identified  along  the 
horizontal  axis  and  the  RIW  rating  criteria  identified 
along  the  vertical  axis.  The  intersection  of  any  combina¬ 
tion  of  these  two  variables  provides  the  ranking  that  one 
participant  gave  to  a  criterion. 

Round  II  Analysis 

As  stated  previously  in  the  methodology,  the  Delphi 
process  requires  some  measure  of  central  tendency  among  the 
respondents  as  an  indicator  of  progress  toward  consensus. 
One  such  measure  of  central  tendency  or  agreement  among  two 
classifications  is  the  Kendall  Tau  statistic  (9:284-294). 
The  Tau  statistic  was  chosen  over  the  Kendall  Coefficient 
of  Concordance — a  test  on  K  sets  of  rankings  instead  of  two 
sets  (9:301) — because  a  comparison  of  pairs  of  rankings 
allowed  the  identification  of  sources  of  non-consensus.  If 
the  Coefficient  of  Concordance  were  used,  it  would  be  pos¬ 
sible  to  test  only  for  the  existence  of  consensus  or  non¬ 
consensus  among  the  group  as  a  whole.  This  test  would  not 
identify  which  participants  disagreed  in  the  case  of  non¬ 
concurrence.  However,  the  comparison  of  pairs  of  rankings 
makes  this  type  of  information  readily  available. 

The  data  required  for  this  test  are  N  pairs  (in 


this  case  N  =  11)  of  observations  where  each  set  is  measured 


on  at  least  an  ordinal  scale.  According  to  Gibbons,  the 
ranking  of  data  within  each  set  serves  this  purpose  (9:284). 
The  test  statistic  for  this  procedure  is 


S  is  computed  by  comparing  the  number  of  pairs  of  rankings 
of  a  second  respondent  which  are  also  arranged  in  their 
correct  or  natural  order  when  they  are  sorted  according  to 
the  natural  order  of  the  rankings  of  the  first  respondent. 

To  compute  S  begin  with  the  observation  ranked  1  by  the 
first  respondent  and  count  the  number  of  ranks  by  the 
second  respondent  which  are  greater  than  the  rank  of  that 
criterion  by  the  second  respondent.  Once  this  has  been 
done,  the  number  of  ranks  below  this  observation  which  are 
smaller  than  its  rank  by  the  second  respondent  are  sub¬ 
tracted  from  the  first  quantity.  The  sum  of  these 
remainders  is  equal  to  S.  S  is  then  divided  by  the  maxi¬ 
mum  possible  S  which  could  have  been  obtained  with  that 
number  of  rankings  had  the  two  sets  of  rankings  been  in 
total  agreement.  This  number  can  be  expressed  as  (1/2)N(N-1) 
where  N  is  the  number  of  observations  or  cases — 11  (15:290). 

The  SPSS  subprogram  NONPAR  CORR  was  used  to  compute 
and  test  the  significance  of  the  Tau  statistic  for  all 
possible  pairs  of  comparisons  of  respondents'  rankings 


Results  of  this  computation  are  contained  in 
Appendix  L.  The  Tau  statistic  is  the  value  contained  in 
the  top  portion  of  the  block  while  the  corresponding  level 
of  significance  of  that  statistical  value  is  displayed  in 
the  lower  portion  of  the  block.  For  example,  to  find  the 
outcome  of  comparing  the  ranking  of  interviewee  4  with 
interviewee  6,  enter  the  table  on  the  left  at  interviewee  4 
and  move  across  to  the  entry  in  the  interviewee  6  column. 
The  Tau  value  is  found  to  be  0.6727  along  with  its  corre¬ 
sponding  level  of  significance  of  0.002. 

The  closer  Tau  is  to  +1,  the  more  the  two  sets  of 
ranks  are  positively  correlated.  Since  all  of  the  Tau 
values  in  Appendix  L  are  positive,  all  of  the  rankings  are 
positively  correlated. 

The  level  of  significance  displayed  in  the  lower 
portion  of  the  block  is  based  on  the  relative  proximity  of 
the  Tau  statistic  to  +1  or  -1.  The  significance  of  Tau  is 
determined  by  comparing  it  to  a  normal  distribution  with  a 
standard  deviation  equal  to 

T4N  +  101 1/2 
_9N  (NT-!)  J 

where  N  *  10  in  this  case  (15:290). 

The  level  of  significance  for  each  Tau  value  can  be 
used  to  test  the  following  hypotheses: 
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H:  No  association  of  the  rankings  exists. 

A+:  The  rankings  are  directly  associated  (9:289). 

Larger  values  of  Tau  result  in  smaller  levels  of 
significance  and  a  rejection  of  the  null  hypothesis.  For 
example,  if  the  association  between  the  rankings  of  inter¬ 
viewees  4  and  6  is  tested  at  the  95  percent  confidence 
level,  (a  =  .05),  reject  the  null  if  the  actual  level  of 
significance  is  less  than  a.  In  other  words,  if  the  prob¬ 
ability  of  rejecting  the  null  hypothesis,  when  in  fact  no 
association  exists,  is  found  to  be  less  than  .05,  the  null 
hypothesis  would  be  rejected,  indicating  that  the  rankings 
are  directly  related.  Since  the  level  of  significance  for 
the  Tau  correlation  between  interviewee  4  and  6  rankings 
(.002)  is  less  than  .05,  it  can  be  concluded,  at  the  95 
percent  confidence  level,  that  the  rankings  of  interviewees 
4  and  6  are  directly  associated. 

Performing  this  analysis  on  all  of  the  correlations 
in  Appendix  L  it  was  concluded  (at  the  95  percent  level  of 
confidence)  that  the  rankings  were  all  associated,  with  the 
exception  of  the  following: 

Interviewee  1  with  interviewee  8 

2  4 

2  10 

3  5 
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Interviewee  3  with  interviewee  7 


3 

3 

4 

5 

7 

8 
8 


8 

10 

8 

8 

8 

9 

10 


If  a  consensus  had  been  reached  in  this  round  of  the 
Delphi  process,  the  null  hypothesis  would  have  been  rejected 
for  each  comparison  of  rankings.  Since  this  was  not  the 
case,  the  Delphi  process  was  continued  with  a  third  round 
of  interviews  providing  anonymous  feedback  from  round  II  to 
the  participants. 


Round  III 

The  third  series  of  interviews  provided  feedback 
to  the  respondents.  A  letter  and  table  of  responses  by  all 
interviewees  is  depicted  in  Appendix  I.  This  information 
was  sent  to  each  participant  prior  to  telephone  contact, 
thereby  allowing  the  respondent  to  reconsider  his/her  rank¬ 
ing  as  compared  to  the  rankings  of  the  other  anonymous 
respondents . 

During  this  round,  each  respondent  was  asked  to 
reconsider  his/her  ranking  of  any  criterion  which  did  not 
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seem  to  agree  with  the  group.  A  response  was  requested  in 
the  form  of  an  explanation  of  extreme  positions  or  a  change 
in  the  rank  order  made  by  the  participant  in  round  II. 

The  results  were  tabulated  as  depicted  in  Appendix 
K.  The  tabulated  data  is  in  the  form  of  a  two  variable 
matrix  in  the  same  format  as  the  tabulation  for  interview 
II. 


Round  III  Analysis 

The  data  in  Appendix  K  was  analyzed  using  the  SPSS 
subprogram  NONPAR  CORR,  as  explained  in  the  analysis  section 
of  round  II  (15:289-292).  Results  of  this  computation  is 
contained  in  Appendix  M  and  is  in  the  same  format  as  the 
analysis  results  for  round  II  contained  in  Appendix  L. 

Following  the  analysis  procedure  for  round  II,  the 
following  set  of  hypotheses  was  tested  using  a  significance 
level  of  0.05: 

H:  No  association  of  the  rankings  exists. 

A+:  The  rankings  are  directly  associated  (9:289). 

Performing  this  analysis  on  all  of  the  correlations 
contained  in  Appendix  M,  it  was  concluded  (at  the  95  percent 
level  of  confidence)  that  the  rankings  were  all  associated, 
with  the  exception  of  the  following: 

Interviewee  3  with  interviewee  6 
3  10 
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Interviewee  4  with  interviewee  5 
4  10 

Since  the  null  hypothesis  was  not  rejected  in  every 
case,  it  could  not  be  concluded  (in  the  strictest  sense  of 
the  methodology)  that  a  consensus  had  been  reached. 

Because  of  time  constraints,  a  fourth  round  was  not 
accomplished.  However,  from  the  data  analysis  accomplished 
to  this  point,  it  is  significant  to  note  the  division  of 
opinion  at  the  end  of  round  III. 

Interviewees  1  through  5  are  program  manager ' s  on 
the  five  programs  sampled  in  this  research,  while  inter¬ 
viewees  6  through  10  are  contracting  officers  for  those 
associated  programs.  With  the  exception  of  the  comparison 
of  the  rankings  of  interviewees  4  and  5,  the  lack  of  con¬ 
sensus  (failure  to  reject  the  null  hypothesis)  occurs  in 
comparisons  between  program  managers  and  contracting 
officers . 

The  level  of  significance  of  the  Tau  statistic 
between  rankings  of  interviewee  4  and  interviewee  5  for 
round  III  is  0.052.  This  level  is  only  slightly  higher 
than  the  selected  alpha  level  of  0.05.  Had  the  selected 
alpha  level  been  only  0.002  higher,  the  null  would  have 
been  rejected  and  it  would  have  been  concluded  that  a 
positive  association  exists  in  the  round  III  rankings  of 
interviewees  4  and  5. 
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The  significance  levels  of  the  other  three  compari¬ 
sons  in  which  the  null  hypothesis  was  not  rejected  are  all 
considerably  higher  than  0.05.  Therefore,  it  can  be  con¬ 
cluded  that  the  significant  breakdown  in  consensus  at  the 
end  of  round  III  was  due  to  differences  in  perception  of 
the  RIW  concept  that  existed  between  contracting  officers 
and  program  managers . 
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CHAPTER  V 


CONCLUSION 

Summary 

Gibbons  suggests  that  when  the  null  hypothesis  of 
no  association  is  rejected,  it  may  be  reasonable  to  conclude 
that  there  exists  a  consensus  with  regard  to  the  ordering 
of  the  objects  and  that  the  observed  results  differ  from  a 
unique  communal  ranking  only  because  of  sampling  variation 
(9:307).  Although  the  null  hypothesis  was  not  rejected  in 
all  comparisons  for  the  entire  group,  it  was  rejected  for 
all  comparisons  of  pairs  within  the  subgroup  of  contracting 
officers  and  the  subgroup  of  program  managers.  Therefore, 
it  was  concluded  that  there  exists  a  consensus  within  the 
group  of  contracting  officers  and  within  the  group  of  pro¬ 
gram  managers  regarding  the  ordering  of  the  RIW  rating 
criteria  nominated  in  round  I.  It  cannot  be  concluded,  how¬ 
ever,  that  a  consensus  exists  within  the  entire  group. 

According  to  Gibbons,  if  it  is  concluded  that  a 
consensus  exists  within  the  group,  then  the  ordering  of 
ranks  for  the  group  is  the  same  as  the  ordering  of  the 
sums  of  ranks  assigned  to  the  objects  (criteria)  (9:307). 
Since  it  was  concluded  that  a  consensus  had  been  reached 
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within  the  subgroups  {contracting  officers  and  program 
managers)  and  ordering  for  each  of  these  subgroups  can  be 
determined  from  the  ranking  results  of  round  III.  From 
Appendix  K,  the  rankings  for  the  two  subgroups  were: 


Proqram  Manager's  Rankinq 

Contracting  Officer's  Rankinq 

1. 

Reliability 

1. 

Reliability 

2. 

Maintainability 

2. 

Acquisition  Cost 

3. 

Acquisition  Cost 

3. 

Maintainability 

4. 

Equipment  Modification 

4. 

Equipment  Modification 

5. 

RIW  Contract  Clause 

5. 

Contractor  Competition 

6. 

Data  Collection 

6. 

Data  Collection 

7. 

Contractor  Competition 

7. 

RIW  Contract  Clause 

Use  Rate  (Tie) 

8. 

Use  Rate 

9. 

Effect  of  RIW  on  Supply 

9. 

Effect  of  RIW  on  Supply 

System 

System 

10. 

Maintenance  Skill 

10. 

Number  of  Spares  Required 

Required 

11. 

Maintenance  Skill  Required 

11. 

Number  of  Spares 

Required 

Between  the  two  subgroups  there  exist  differences  in 
the  ranking  of  several  criteria.  According  to  comments 
received  from  round  III  interviews,  the  following  explana¬ 
tions  were  provided.  RIW  clause  was  perceived  to  be  less 
important  by  contracting  officers  because  "the  loopholes  in 
contracts  prevent  the  program  from  working  as  designed  and 
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make  it  difficult  to  determine  the  actual  purpose  of  RIW." 
One  contracting  officer  cited  this  explanation  for  his  low 
ranking  of  RIW  clause:  "The  author  of  the  contract  is  pro¬ 
vided  guidelines  from  the  System  Project  Office  {SPO) ;  some¬ 
times  these  guidelines  are  not  passed  on  to  other  members 
working  in  the  contracting  office.  The  result  is  simple — 
we  as  contractors  don't  operate  with  the  same  common  body 
of  information  resulting  in  confusion  of  the  intent  of 
RIW.  " 

The  program  managers  provided  the  following  explana¬ 
tion  of  their  higher  rating  of  RIW  clause.  "A  RIW  program 
is  most  effective  when  a  MTBF  guarantee  is  included  in  the 
contract.  Our  program  has  such  a  clause  and  it  makes  it 
easier  to  administer  items  that  don't  meet  MTBF  require¬ 
ments. 

Another  criterion  the  two  subgroups  ranked  dif¬ 
ferently  was  contractor  competition.  Contracting  officers 
ranked  the  criterion  as  being  more  important.  One  con¬ 
tracting  officer  stated,  "...  must  have  basic  contractor 
competition  before  even  considering  an  RIW  application. 

One  of  the  basic  guidelines  for  RIW  is  contractor  competi¬ 
tion — without  it,  the  concept  of  an  improved  warranty  will 
certainly  fail."  Conversely,  a  program  manager  remarked 
that  "contractor  competition  has  very  little  impact  on 
assessing  the  effectiveness  of  RIW." 


The  preceding  explanations  account  for  some  of  the 
differences  between  program  managers  and  contracting 
officers.  These  polarized  positions  indicate  that  the 
respondents  feel  strongly  in  their  ranking  of  RIW  clause 
and  contractor  competition. 

The  above  analysis  results  and  conclusions  can  now 
be  related  to  the  research  ob jective--to  determine  a  set  of 
criteria  from  among  the  current  RIW  programs  that  can  be 
incorporated  into  a  plan  for  evaluating  the  RIW  concept. 
During  round  I,  eleven  criteria  were  nominated  and  presented 
to  the  participants  for  their  consideration.  Based  upon 
the  analysis  of  round  I  results,  four  criteria  were  elimi¬ 
nated  from  the  original  list  and  four  were  added.  Subse¬ 
quent  rounds  did  not  result  in  any  further  additions  or 
deletions  to  the  list.  The  rank-ordered  lists  on  page  34 
contain  the  criteria  which  the  participants,  as  a  group, 
believe  to  be  important  to  any  plan  for  evaluation  of  the 
RIW  concept.  It  is  important  to  note,  however,  that  this 
list  is  based  solely  on  the  perceptions  of  contracting 
officers  and  program  managers  within  five  system  program 
offices . 

The  answer  to  research  question  1 — elements  affected 
by  RIW  in  Air  Force  programs  that  use  it — consists  of  the 
list  of  criteria.  Research  question  2  is  answered  in  the 
rank  order  placed  on  this  list  of  criteria  by  the  two 
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subgroups.  It  is  important  to  note  that  complete  agreement 
of  the  rankings  was  not  established  for  the  group  as  a 
whole.  Only  if  the  rank-ordered  lists  of  each  subgroup  are 
considered  separately  can  question  2  be  addressed.  There¬ 
fore,  based  upon  the  analysis,  a  consensus  of  opinion  on 
research  question  2  was  not  reached. 

Recommended  Research  Areas 

Opinions  by  the  group  of  RIW  experts  sampled  in 
this  study  raises  questions  regarding  the  effect  that  the 
individual's  job  has  on  his  perception  of  important  rating 

9 

criteria  for  the  RIW  concept. 

1.  Since  this  research  effort  sampled  only  con¬ 
tracting  officers  and  program  managers,  the  population,  as 
outlined  in  Chapter  II,  should  be  examined  further.  For 
example,  contractors,  design  engineers,  field  maintenance 
personnel,  etc.,  could  be  sampled  to  determine  whether 
their  perceptions  are  comparable  to  the  contracting  officers 
and  program  managers,  and  to  each  other. 

2.  As  described  in  Chapter  I,  a  need  exists  to 
establish  a  plan  for  data  collection.  Therefore,  based  on 
the  evaluation  criteria  listed  in  the  summary  of  this  chap¬ 
ter,  the  feasibility  of  data  collection  for  these  evaluation 
criteria  should  be  addressed. 
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APPENDICES 


Warranty 


Program 

Contractor 

Start  Dt. 

Period 

ARN  118  TACAN 

Collins  Avionics  Group, 

Dec.  7 5/ 

5  years 

Rockwell  International 

Apr . 7  7 

C-141/KC-135/C-5 

Delco  Electronics, 

Mar. 77 

4  years 

Inertial  Navigation 
System 

General  Motors  Corp.  | 

OMEGA  Navigation  Set 

Dynell  Electronics  Corp. 

Mar . 77 

5  years 

F-16  Aircraft  Components 

General  Dynamics 

Jan. 79 

4  years 

Flight  Control  Computer 

Lear  Siegler,  Inc.  (Sub.) 

or 

Radar  Antenna 

Westinghouse  (Sub.) 

300,000 

Radar  Low  Power  RF 

Westinghouse  (Sub.) 

flying 

Radar  Digital  Processor 

Westinghouse  (Sub.) 

hours 

Radar  Computer 

Westinghouse  (Sub.) 

(which- 

Heads  Up  Display  Unit 

Marconi-Elliott  (Sub.) 

ever 

(HUD) 

comes 

Navigation  Unit 

Singer  (Sub.) 

first) 

Radar  Transmitter 

Westinghouse  (Sub.) 

HUD  Electronics 

Marconi  Elliott  (Sub.) 

C-141  Attitude 

Heading  Reference 
System 


Lear  Siegler,  Inc. 


Oct. 75 


5  years 


APPENDIX  B 

OTHER  AIR  FORCE  APPLICATIONS  OF  RIW  (12) 


41 


? 

i 

'f 
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Program 

F-lll  Displacement  Gyro 
AV  -8  C/A  Airspeed  Indicator 
C-130  Hydraulic  Pump 
Klystron  Electron  Tubes 


Contractor 
Lear  Siegler,  Inc. 
Bendix  Corp. 

Abex  Corp. 


Varian  Associates 


APPENDIX  C 
DELPHI  TECHNIQUE  ( 


1:53) 
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Start 
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APPENDIX  D 

RESPONSIBILITY  CENTERS  FOR  RIW  (12) 
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Current  Responsibilities 

Original  Responsible 

Contracting 

System 

Office 

Office 

Manager  , 

* 

★  ■ 

TACAN 

ESD/OCN 

WR-ALC/PM 

WR-ALC/MMIM 

Dec.  75/Apr.  77 

★ 

* 

AHRS 

ASD/AEA 

OC-ALC/PM 

OC-ALC/MMIM 

Oct.  75 

j 

★ 

* 

OMEGA 

ASD./AEA 

ASD/AEA 

WR-ALC/MMIM 

Mar.  77 

* 

* 

INS 

WR/ALC/PM  (Contract) 

WR-ALC/PM 

OC-ALC/MMIM 

Mar.  77 

UR-ALC/MME  (RIW) 

OC-ALC/MMIM  (Program 

Manager 

F-16 

ASD/YPK  (Contractor) 

Jan.  79 

ASD/YPL  (Program 

Manager) 

*  Those  offices  that  comprise  the  research  sample. 


46 


APPENDIX  E 


LETTER/INTERVIEW  I 


DEPARTMENT  OF  THE  AIR  FORCE 

a:!?  f crce  institute  of  TECHNOLOGY  ■  ATC ; 
WRIGHT  PATTERSON  AIR  FORCE  3ASE.  OH'C  45433 


LSY-1  (LS6R  63-80/Capt  J.  dradney/Capt  M.  Perkins/ 
AUTOVON  785-6569) 

Reliability  Improvement  Warranty  (RIW)  Concept  Interview 


1.  The  attached  interview  plan  was  prepared  by  a  research  team  at  the 
air  Force  Institute  of  Technology,  Wright  Patterson  .-^3,  OH.  The  inter¬ 
view  will  be  the  first  in  a  series  of  three  or  four.  The  series  will 

be  conducted  using  the  delphi  Technique,  a  procedure  designed  to  deter¬ 
mine  successful  group  decisions  when  the  scope  of  the  problem  is  such 
that  no  one  individual  possesses  all  of  the  knowledge  and  expertise  to 
effect  a  solution. 

2.  The  managerial  position  you  hold  within  your  program  identifies  you 
to  be  among  the  most  knowledgeable  on  the  Air  Force  experience  with  t\I.;. 
It  is  anticipated  that  your  responses,  coupled  with  the  experience  gained 
in  otaer  RIW  applications,  will  shed  new  light  on  the  impact  ul  ,'s  have 
had  in  the  air  Force. 

3.  Your  responses  in  all  interviews  will  be  kept  completely  anonymous 
both  within  the  group  of  interviewees,  and  in  tne  report,  and,  of  course, 
your  participation  is  voluntary. 


4.  Capt  Jradney  or  Oapt  Perkins  will  be  contacting  you  by  telephone  in 
a  few  days.  Your  cooperation  in  providing  this  data  will  be  greatly 
appreciated  and  very  beneficial  in  advancing  the  documented  knowlaoue 
about  tue  impact  of  RIWs  within  the  Air  Force. 


IxkiLId  Z/vHiiO,  Major,  USAF 

Assistant  t^ofessor  of  Quantitative  Methods 


Thesis  .\dvisor 


1  Atch 

Interview  Plan 
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The  purpose  of  this  research  is  to  determine  what, 
if  any,  characteristics  about  Air  Force  RIW  applications 
might  be  used  to  evaluate  the  overall  effectiveness  of  RIW 
use  within  the  Air  Force. 

The  first  interview  will  contain  two  parts.  In  the 

first  part,  you  will  be  asked  for  some  information  on  the 

following  selected  characteristics : 

Contractor  competition 
Acquisition  cost 
Equipment  limitation 
Initial  support  cost 
Logistics  support  cost 
Reliability 
Maintainability 
Number  of  spares  required 
Maintenance  skill  required 
Support  manpower 
Equipment  modification 

The  following  information  will  be  requested  about 
each  of  the  characteristics  listed  above: 

1.  Do  you  feel  that  RIW  has  had  a  significant 
impact  (good  or  bad)  on  this  item? 

2.  Can  you  think  of  a  standard  (i.e.,  non-RIW 
application)  to  compare  this  impact  with? 

3.  Is  any  information  (data)  related  to  this  ele¬ 
ment  being  collected  in  your  program? 

The  second  part  of  the  interview  will  ask  you  to 
identify  any  additional  elements  which  you  feel  would  be 


important  in  assessing  the  impact  of  RIW  on  your  program. 
In  regard  to  these  additional  characteristics,  the  follow¬ 
ing  two  questions  will  be  asked: 


1.  Can  you  think  of  a  standard  against  which  you 
could  compare  the  impact  of  RIW  on  this  item? 

2.  Is  information  related  to  this  element  being 
collected  in  your  program? 
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APPENDIX  F 
INTERVIEW  I  GUIDE 


As  the  letter  you  received  a  few  days  ago  indicated, 


the  purpose  of  this  research  is  to  determine  what,  if  any, 
characteristics  about  Air  Force  RIW  applications  might  be 
used  to  evaluate  the  overall  effectiveness  of  RIW  use  within 
the  Air  Force.  This  is  the  first  in  a  series  of  interviews 
to  collect  information  about  your  program.  Because  of  your 
experience  with  RIW,  we  anticipate  that  your  responses  will 
be  extremely  helpful  in  this  research  effort. 

Today's  interview  contains  two  parts.  In  the  first 
part,  I  will  ask  you  for  some  information  on  the  applicabil¬ 
ity  of  selected  characteristics  to  your  program.  In  the 
second  part,  I  will  ask  you  to  identify  any  other  character¬ 
istics  which,  in  your  experience,  would  be  useful  to  assess 
the  impact  of  RIW  on  your  program. 

The  first  characteristic  is  contractor  competition. 

1.  Do  you  feel  that  RIW  has  had  a  significant 
impact  (good  or  bad)  on  this  item? 

2.  Can  you  think  of  a  standard  (i.e.,  non-RIW 
application)  to  compare  this  impact  with? 

3.  Is  any  information  (data)  related  to  this  ele¬ 
ment  being  collected  in  your  program? 

(Continue  the  interview  by  repeating  the  above 
three  questions  for  each  of  the  remaining  items.) 
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Acquisition  cost 
Equipment  limitation 
Initial  support  cost 
Logistics  support  cost 
Reliability 
Maintainability 
Number  of  spares  required 
Maintenance  skill  required 
Support  manpower 
Equipment  modification 

If  you  were  asked  to  assess  the  impact  of  RIW  in 
this  program,  are  there  additional  elements  which  you  would 
use  in  the  assessment?  (If  the  respondent  replies  with 
additional  items,  ask  him  for  the  following  information 
about  each  one . ) 

9 

1.  Can  you  think  of  a  standard  against  which  you 
could  compare  the  RIW  impact  on  this  item? 

2,  Is  information  related  to  this  element  being 
collected  in  your  program? 

This  completes  the  interview.  The  information  you 
have  provided  will  be  a  great  help  in  this  research  effort. 

It  will  be  compiled  with  information  gathered  about  other 
Air  Force  RIW  applications  to  modify  this  preliminary  list 
of  criteria  we  have  discussed  today.  I  will  contact  you 
again  in  a  few  days  to  ask  your  opinion  about  this  new  list. 

Once  again,  thank  you  for  your  patient  participation. 
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DEPARTMENT  OF  THE  AIR  FORCE 
AIR  FORCE  INSTITUTE  OF  TECHNOLOGY  ■  ATC : 
WRIGHT  PATTERSON  AIR  FORCE  BASE.  OHIO  45433 


LSY-1  (LS5R  63-80/Capt  J.  Bradney/Capt  M.  Perkins/ 

AUTOVON  785-6569) 

Reliability  Improvement  Warranty  (RIW)  Concept  Interview  II 


1.  The  purpose  of  this  portion  of  the  survey  is  to  determine  cur¬ 
rent  attitudes  toward  criteria  (that  you  have  identified)  that  may 
be  incorporated  into  an  evaluation  plan  of  RIW.  The  results  of 
this  survey  will  be  written  up  into  a  formal  research  paper  in 
partial  fulfillment  of  the  requirements  for  a  Master  of  Science 
Degree.  Your  support  and  effort  will  be  greatly  appreciated. 

2.  The  enclosed  survey  should  require  only  several  minutes  to 
complete.  Results  of  this  survey  may  provide  guidance  in  a  formal 
evaluation  of  RIW. 

3.  The  survey  is  designed  to  test  opinions  of  those  persons  who 
have  dealt  with  Rl.is.  All  individuals  participating  are  either 
contracting  officers  or  corporate  project  managers  for  an  itemis; 
under  KlW. 

4.  Your  responses  to  the  survey  will  be  requested  by  telephone  in 
a  few  days  after  your  receipt  of  this  letter. 

5.  Names  of  individuals  participating  will  be  anonymous  to  one  an¬ 
other  and  will  not  be  cited  in  the  research  report.  Tne  informa¬ 
tion  you  have  orovided  will  be  of  great  assistance  in  this  research 
effort.  Thank  you  for  your  cooperation  and  time. 


LESLIE  J.  ZAMBO,  Major,  USaF  '  1  Atch 

Assistant  Professor  of  Quantitative  Methods  Interview  Plan 

Thesis  Advisor 
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During  interview  I,  you  were  asked  to  identify  RIW 
characteristics  applicable  to  your  program.  Based  on  your 
responses  and  those  of  other  representatives  of  programs 
that  employ  RIW,  the  following  alphabetized  list  of  char¬ 
acteristics  has  been  compiled.  This  list  consists  of  char¬ 
acteristics  which  might  be  used  to  evaluate  the  overall 
effectiveness  of  RIW  within  the  Air  Force. 

If  you  were  in  a  position  to  assess  the  overall 
effectiveness  of  RIW  in  your  program,  how  would  you  rank 
each  of  the  characteristics  below?  Please  rank  the  follow¬ 
ing  characteristics  in  order  (1  through  11)  beginning  with 
the  best  characteristic  as  "1".  If  any  characteristic  is 
not  applicable  to  your  program,  write  "NA"  in  the  space 
provided.  Please  include  any  comments  you  feel  are  neces¬ 
sary  for  clarification. 

_  Acquisition  cost 

Comment : 

_  Contractor  competition 

Comment : 


Data  collection  (the  need  for  a  standard  set  of  data 
to  perform  analysis  of  equipment  under  an  RIW) 
Comment : 

Effect  of  RIW  on  supply  system  (Since  most  RIW  items 
are  returned  to  the  contractor  for  repair,  does  this 
impact  the  supply  system?) 

Comment : 
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Equipment  modification  (improvements  from  engineering 
change  proposals) 

Comment: 


Maintainability 
Comment:  ~  “  ** 


Maintenance  skill  required  (impact  of  the  RIW  clause 
on  maintenance  handling  procedures  of  failed  RIW 
items) 

Comment: 


Number  of  spares  required  (consignment) 
Comment: 


Reliability 
Comment : 


RIW  contract  clause  (Do  "loopholes"  in  the  RIW  clause 
prevent  the  warranty  from  accomplishing  its  purpose?) 
Comment : 


Use  rate  (This  characteristic  refers  to  the  impact 
equipment  usage — i.e.,  reduced  flying  hours  due  to 
fuel  costs — has  on  the  effectiveness  of  RIW.) 
Comment: 
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DEPARTMENT  OF  THE  AIR  FORCE 
AIR  FORCE  INSTITUTE  OF  TECHNOLOGY  '  ATC  : 
WEIGHT  PATTSRSON  AIR  FORCE  BASE.  OHIO  45433 


cplv  ro 

t-  or  LSY-I  (LSiR  63-80/Capt  J.  Qradney/Capt  M.  Perkins/ 

AUTOVON  785-6569) 

Reliability  Improvement  Warranty  (RIW)  Concept  Interview  III 


TC 


1.  During  interview  II,  you  were  asked  to  rank  an  alphabetized  list  of 
characteristics  common  to  existing  RIW  programs.  The  results  of  inter¬ 
view  II  are  provided  in  the  following  attachment,  four  responses  are 
identified  by  a  red  circle  around  the  appropriate  interviewee  number 
(i.e.  if  you  were  interviewee  4,  there  appears  a  red  circle  around  ”4” 
and  your  results  are  in  column  4). 

2.  At  this  time,  you  are  requested  to  reevaluate  your  current  ranking 
of  the  characteristics  based  upon  the  opinions  provided  by  the  other 
interviewees.  The  purpose  of  this  information  feedback  is  to  provide 
you  with  new  data,  in  the  form  of  other  expert  opinions,  to  determine 

if  extreme  positions  are  properly  identified.  For  example,  if  the  charac¬ 
teristic,  RIW  contract  clause  is  ranked  6,  7,  8,  or  9  by  otner  experts 
and  your  perception  was  that  the  RIW  contract  clause  was  extremely  im¬ 
portant,  and  your  subsequent  ranking  resulted  in  1  or  2  then  either 
provide  reason  for  your  extreme  position  or  rerank  this  characteric, 
whichever  is  appropriate. 


LSSLIK  J.  ZAMBO,  Major,  USaF  2  Attachments 

Assistant  Professor  of  Quantitative  Methods  1.  Interview  II  results 

Thesis  Advisor  2»  Interview  III  plan 
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APPENDIX  K 
RESULTS  OF  ROUND  III 


APPENDIX  L 


KENDALL  CORRELATION  TEST  RESULTS 
FOR  ROUND  II 
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APPENDIX  M 
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